Image display system, image display apparatus, and control method thereof

ABSTRACT

Upon receiving a communication switching instruction from a first wireless access point used for communication with an image processing apparatus, an image display apparatus disconnects communication with the first wireless access point. Simultaneously, the image display apparatus transmits, to a second wireless access point, a link request to establish communication with the second wireless access point of a new communication destination included in the switching instruction. The image display apparatus displays, on a display unit, a captured image continuously acquired from am image capturing unit until switching from the first wireless access point to the second wireless access point finishes as communication destination switching.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image display system which causes animage display apparatus to receive an image from an image processingapparatus via a wireless access point and display the image, an imagedisplay apparatus, and a control method thereof.

2. Description of the Related Art

Recently, so-called MR (Mixed Reality) techniques are known astechniques of seamlessly combining a physical world and a virtual worldin real time. A known MR technique uses a video see-through Head MountedDisplay (HMD). An image capturing apparatus captures an image of anobject which almost matches an object viewed from the pupil position ofan HMD user. The HMD user can view an MR image obtained by superimposinga Computer Graphics (CG) image on the captured image.

FIG. 10 is a functional block diagram of a general video see-throughmixed reality system which wirelessly transmits an image. An overview ofthe operation will be described with reference to FIG. 10.

An image display apparatus 1001 is, for example, a video see-throughHMD. The image display apparatus 1001 has an image capturing unit 1004,three-dimensional position and orientation measurement sensor 1005,wireless communication I/F 1006, and display unit 1007.

The image capturing unit 1004 captures an external observation imagewhich almost matches the line-of-sight position of the HMD user. Theimage capturing unit 1004 includes two sets of image capturing elementsand optical systems for the right and left eyes, which generate astereoscopic image, and a signal processing circuit, such as a DigitalSignal Processing (DSP) circuit, for performing image processing of thesucceeding stage.

The three-dimensional position and orientation measurement sensor 1005measures the three-dimensional position and orientation of a measurementtarget to calculate the rendering position of a CG image. For example,the three-dimensional position and orientation measurement sensor 1005acquires the three-dimensional position and orientation position (to bereferred to as position and orientation information hereinafter) of theHMD user (image display apparatus 1001). The three-dimensional positionand orientation measurement sensor 1005 is implemented by a magneticsensor or a gyro sensor (acceleration and angular velocity).

The wireless communication I/F 1006 transmits or receives data to orfrom a wireless access point 1002 a or 1002 b. The wirelesscommunication I/F 1006 transmits, for example, the image captured by theimage capturing unit 1004 and the position and orientation informationto the wireless access point 1002 a together and receives a compositedMR image. The wireless communication I/F 1006 needs to perform real-timeprocessing and uses a high-speed wireless standard such as UWB or IEEE802.11n capable of high-bandwidth transmission.

The display unit 1007 displays an MR image with a superimposed CG image.The display unit 1007 includes two sets of display devices and opticalsystems for the right and left eyes. As the display device, a smallliquid crystal display or a retina scan type device utilizing MEMS isused.

Each of the wireless access points (AP) 1002 a and 1002 b performsinformation transmission with respect to the image display apparatus1001 using wireless communication. More specifically, each of thewireless access points (AP) 1002 a and 1002 b transmits a display imagethat is a composite image with a captured image to the image displayapparatus 1001, or receives output information from thethree-dimensional position and orientation measurement sensor 1005 usingwireless communication.

Reference numeral 1003 denotes an image processing apparatus. The imageprocessing apparatus 1003 renders a CG image based on the captured imageand the position and orientation information received from the imagedisplay apparatus 1001 and composites it with the captured image. Theimage processing apparatus 1003 is generally implemented by an apparatushaving advanced arithmetic processing function and graphic displayfunction, such as a personal computer or a workstation.

The image processing apparatus 1003 has a communication I/F 1008,position and orientation information generation unit 1009, CG renderingunit 1010, content storage unit 1011, and image composition unit 1012.

The communication I/F 1008 is the communication I/F on the side of theimage processing apparatus 1003. The communication I/F 1008 needs toperform real-time processing and uses a metal wire such as a USB or IEEE1394 or an optical fiber such as GigabitEthernet.

The position and orientation information generation unit 1009 generatesthe position and orientation information of the wearer of the imagedisplay apparatus 1001 based on the received captured image and positionand orientation information. In addition to the information from thethree-dimensional position and orientation measurement sensor 1005 ofthe image display apparatus 1001, a marker may be extracted from thecaptured image and used as correction information.

The CG rendering unit 1010 renders a CG image based on the position andorientation information and a content acquired from the content storageunit 1011. The content storage unit 1011 stores contents for generatinga virtual image in virtual space.

The image composition unit 1012 composites the received captured imageand the virtual image generated by the CG rendering unit 1010. The imagecomposition unit 1012 transmits the obtained MR image (composite image)to the wireless access point 1002 a or 1002 b via the communication I/F1008. The wireless access point 1002 a or 1002 b transmits the MR imageto the image display apparatus 1001 by wireless communication. The MRimage is thus displayed on the display unit 1007 of the image displayapparatus 1001.

The above-described arrangement and processing allow any user who wearsthe video see-through HMD to experience a mixed reality world in which aphysical world and a virtual world are seamlessly combined in real time.

Japanese Patent Application Laid-Open No. 11-88913 (FIG. 7, paragraph0035) and Kato, H., Billinghurst, M., “Marker Tracking and HMDCalibration for a video-based Augmented Reality Conferencing System”, InProceedings of the 2nd International Workshop on Augmented Reality (IWAR99), San Francisco, USA, October, (1999) (hereinafter, “Marker Trackingand HMD Calibration for a video-based Augmented Reality ConferencingSystem”) disclose the arrangements of general MR techniques and systems.

The concept of position and orientation information generation using amarker will be described with reference to FIG. 11.

Referring to FIG. 11, the positional relationship between a marker 1103and the image capturing apparatus is defined in advance. When the marker1103 is displayed in a physical space image 1101, the position andorientation measurement unit detects the marker 1103 from the imagedata. It is possible to calculate the relative positional relationshipbetween the marker 1103 and the image capturing apparatus main body andposition and orientation information in the marker observation directionof the HMD user on the basis of information such as the size, shape, andfill pattern of the detected marker 1103.

FIG. 11 assumes a three-dimensional coordinate system (X axis 1105 a, Yaxis 1105 b and Z axis 1105 c) having its origin at the center of themarker 1103. However, the origin of the coordinate system need notalways be set on the marker 1103. The origin of the coordinate systemcan be set at an arbitrary position by defining the relative positionalrelationship between it and the marker 1103. A plurality of markers maybe used simultaneously for position and orientation informationgeneration. To use a plurality of markers simultaneously, the positionalrelationship between the markers is defined in advance. This allowscalculation of the marker observation direction based on the relativepositional relationship between the markers.

It is therefore possible to use not the marker 1103 which has aninternal fill pattern to identify the direction, as shown in FIG. 11,but, for example a color marker or a marker formed from a light-emittingelement such as an LED which has no direction information butone-dimensional information. Not the marker 1103 but a feature point inthe image such as an outline 1104 of a table 1102 or a specific color inthe image may be extracted and used to calculate position andorientation information. It is also possible to generate more accurateposition information by using a plurality of markers of the same type ora plurality of kinds of markers simultaneously, or combining markerinformation and the information of feature points in the image. Thepositional relationship between the plurality of markers or featurepoints is defined in advance. Hence, even when not all markers orfeature points are displayed in the image, the position of each markeror feature point can be estimated.

“Marker Tracking and HMD Calibration for a video-based Augmented RealityConferencing System” discloses implementation of an MR technique usingmarkers.

In the system shown in FIG. 10, when the HMD user moves over a widearea, switching of a plurality of wireless access points is necessary.To allow for switching (handover) of the wireless access points, aplurality of antennas are provided on a mobile terminal itself. Beforearea switching, the mobile terminal establishes a link with the basestation in the switching destination and then disconnects the precedinglink so that it can always wirelessly communicate with any one of basestations. Japanese Patent Application Laid-Open No. 10-16464 (FIG. 1)(hereinafter, “JPLO 10-16464”) discloses an example of this arrangement.

JPLO 10-16464 discloses an arrangement which performs area switching ina range where the communication range of a base station of a currentlink overlaps that of a base station of the new link establishmenttarget.

Japanese Patent Application Laid-Open No. 5-76078 (hereinafter, “JPLO5-76078”) discloses a method of interpolating an unreceived image whenwireless image transmission has been interrupted due to some reason.

In the arrangement disclosed in JPLO 5-76078, when infrared rays usedfor wireless transmission are intercepted for some reason, a signal of astill image, which is prepared in advance, is displayed.

However, the above-described prior arts have the following problems.

If an HMD having a wireless communication function moves over a widearea, the time required for handover is nonnegligible for the HMD user.Hence, images are interrupted until the completion of handover, that is,the time from disconnection of a link to establishment of a new link, orduring the time from establishment of a new link to reception of adisplay image.

In the arrangement for always keeping at least one link established byswitching a plurality of antennas, the mobile terminal must have aplurality of antennas and a plurality of circuits for communication.This leads to a bulky apparatus arrangement or circuit scale.Additionally, since a plurality of link states occur simultaneously,band assignment for communication is necessary. This limits the usableband in the whole space and limits the number of HMDs that use a singlewireless access point. Furthermore, if a specific still image isdisplayed assuming interruption, it obstructs the HMD user's view. Thisposes a problem of safety in a system assuming movement.

SUMMARY OF THE INVENTION

The present invention has been made to solve the above-describedproblems. The present invention provides an image display system capableof providing a mixed reality environment to an observer without anysense of incongruity even upon switching wireless access points, animage display apparatus, a control method thereof, and a computerprogram.

According to an aspect of the present invention, there is provided animage display system including: an image display apparatus, an imageprocessing apparatus and a plurality of wireless access points. Theimage display apparatus includes: an image capturing unit which capturesan image of physical space; a wireless communication unit whichcommunicates with the image processing apparatus, including transmissionof the captured image obtained by the image capturing unit to the imageprocessing apparatus, and a display unit which receives, from the imageprocessing apparatus, a composite image of the captured image and avirtual image composited by the image processing apparatus and displaysthe composite image. The image processing apparatus includes: ageneration unit which generates position and orientation information ofthe image display apparatus; a virtual image generation unit whichgenerates the virtual image based on the position and orientationinformation, a composition unit which composites the captured image andthe virtual image; and a communication unit which communicates with theimage display apparatus, including transmission of the composite imageobtained by the composition unit to the image display apparatus. Theplurality of wireless access points are used for communication betweenthe image display apparatus and the image processing apparatus. Theimage processing apparatus also includes: a determination unit thatdetermines, based on a position of the image display apparatus, whetherswitching from a first wireless access point used for communication withthe image display apparatus to a second wireless access point isnecessary; and a switching instruction unit that, if the determinationunit determines that the switching is necessary, transmits a switchinginstruction to the first wireless access point and the second wirelessaccess point. The image display apparatus also includes: a communicationcontrol unit that, upon receiving the communication switchinginstruction from the first wireless access point used for communicationwith the image processing apparatus, disconnects communication with thefirst wireless access point and transmits, to the second wireless accesspoint, a link request to establish communication with the secondwireless access point of a new communication destination included in theswitching instruction; and a display control unit that displays, on thedisplay unit, the captured image continuously acquired from the imagecapturing unit until switching from the first wireless access point tothe second wireless access point finishes as communication destinationswitching.

According to another aspect of the present invention, there is providedan image display system including an image display apparatus, an imageprocessing apparatus and a plurality of wireless access points. Theimage display apparatus includes: an image capturing unit which capturesan image of physical space, a wireless communication unit whichcommunicates with the image processing apparatus, including transmissionof the captured image obtained by the image capturing unit to the imageprocessing apparatus, a display unit which receives, from the imageprocessing apparatus, a composite image of the captured image and avirtual image composited by the image processing apparatus and displaysthe composite image, and an image superimposition display unit whichsuperimposes a message image on a display image of the display unit. Theimage processing apparatus includes: a generation unit which generatesposition and orientation information of the image display apparatus, avirtual image generation unit which generates the virtual image based onthe position and orientation information, a composition unit whichcomposites the captured image and the virtual image, and a communicationunit which communicates with the image display apparatus, includingtransmission of the composite image obtained by the composition unit tothe image display apparatus. The plurality of wireless access points areused for communication between the image display apparatus and the imageprocessing apparatus. The image processing apparatus also includes: adetermination unit that determines, based on a position of the imagedisplay apparatus, whether switching from a first wireless access pointused for communication with the image display apparatus to a secondwireless access point is necessary; and a switching instruction unitthat, if the determination unit determines that the switching isnecessary, transmits a switching instruction to the first wirelessaccess point and the second wireless access point. The image displayapparatus includes: a communication control unit that, upon receivingthe communication switching instruction from the first wireless accesspoint used for communication with the image processing apparatus,disconnects communication with the first wireless access point andtransmits, to the second wireless access point, a link request toestablish communication with the second wireless access point of a newcommunication destination included in the switching instruction; and adisplay control unit that causes the image superimposition display unitto superimpose the message image on the display image of the displayunit until switching from the first wireless access point to the secondwireless access point finishes as communication destination switching.

According to another aspect of the present invention, there is providedan image display system including: an image display apparatus, an imageprocessing apparatus, and plurality of wireless access points. The imagedisplay apparatus includes: an image capturing unit which captures animage of physical space, a storage unit which stores a virtual image, acomposition unit which composites the captured image captured by theimage capturing unit and the virtual image stored in the storage unit, adisplay unit which displays a composite image composited by thecomposition unit, and a wireless communication unit which communicateswith the image processing apparatus, including reception of the virtualimage from the image processing apparatus. The image processingapparatus includes: a virtual image generation unit which generates thevirtual image based on position and orientation information of the imagedisplay apparatus; and a communication unit which communicates with theimage display apparatus, including transmission of the virtual image tothe image display apparatus. The plurality of wireless access points areused for communication between the image display apparatus and the imageprocessing apparatus. The image processing apparatus also includes: adetermination unit that determines, based on a position of the imagedisplay apparatus, whether switching from a first wireless access pointused for communication with the image display apparatus to a secondwireless access point is necessary; and a switching instruction unitthat, if the determination unit determines that the switching isnecessary, transmits a switching instruction to the first wirelessaccess point and the second wireless access point. The image displayapparatus also includes: a communication control unit that, uponreceiving the communication switching instruction from the firstwireless access point used for communication with the image processingapparatus, disconnects communication with the first wireless accesspoint and transmits, to the second wireless access point, a link requestto establish communication with the second wireless access point of anew communication destination included in the switching instruction; anda display control unit that displays, on the display unit, the compositeimage composited by the composition unit using the captured image andthe virtual image received from the image processing apparatusimmediately before the switching and stored in the storage unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

According to another aspect of the present invention, there is providedan image display apparatus including an image capturing unit whichcaptures an image of physical space, and a display unit which receives,from an image processing apparatus, a composite image of a virtual imageand the captured image captured by the image capturing unit, which iscomposited by the image processing apparatus, and displays the compositeimage. The image display apparatus includes: a wireless communicationunit that communicates with the image processing apparatus via one of aplurality of wireless access points; a communication control unit that,upon receiving a communication switching instruction from a firstwireless access point used for communication with the image processingapparatus, disconnects communication with the first wireless accesspoint and transmits, to a second wireless access point, a link requestto establish communication with the second wireless access point of anew communication destination included in the switching instruction; anda display control unit that displays, on the display unit, the capturedimage continuously acquired from the image capturing unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

According to yet another aspect of the present invention, there isprovided an image display apparatus including an image capturing unitwhich captures an image of physical space, a display unit whichreceives, from an image processing apparatus, a composite image of avirtual image and the captured image captured by the image capturingunit, which is composited by the image processing apparatus, anddisplays the composite image, and an image superimposition display unitwhich superimposes a message image on a display image of the displayunit. The image display apparatus includes: a wireless communicationunit that communicates with the image processing apparatus via one of aplurality of wireless access points; a communication control unit that,upon receiving a communication switching instruction from a firstwireless access point used for communication with the image processingapparatus, disconnects communication with the first wireless accesspoint and transmits, to a second wireless access point, a link requestto establish communication with the second wireless access point of anew communication destination included in the switching instruction; anda display control unit that causes the image superimposition displayunit to superimpose the message image on the display image of thedisplay unit until switching from the first wireless access point to thesecond wireless access point finishes as communication destinationswitching.

According to another aspect of the present invention, there is providedan image display apparatus including an image capturing unit whichcaptures an image of physical space, a storage unit which stores avirtual image, a composition unit which composites the captured imagecaptured by the image capturing unit and the virtual image stored in thestorage unit, and a display unit which displays a composite imagecomposited by the composition unit. The image display apparatusincludes: a wireless communication unit that communicates with an imageprocessing apparatus via one of a plurality of wireless access points; acommunication control unit that, upon receiving a communicationswitching instruction from a first wireless access point used forcommunication with the image processing apparatus, disconnectscommunication with the first wireless access point and transmits, to asecond wireless access point, a link request to establish communicationwith the second wireless access point of a new communication destinationincluded in the switching instruction; and a display control unit thatdisplays, on the display unit, the composite image composited by thecomposition unit using the captured image and the virtual image receivedfrom the image processing apparatus immediately before the switching andstored in the storage unit until switching from the first wirelessaccess point to the second wireless access point finishes ascommunication destination switching.

According to yet another aspect of the present invention, there isprovided a method of controlling an image display apparatus including animage capturing unit which captures an image of physical space, adisplay unit which receives, from an image processing apparatus, acomposite image of a virtual image and the captured image captured bythe image capturing unit, which is composited by the image processingapparatus, and displays the composite image, and a wirelesscommunication unit which communicates with the image processingapparatus via one of a plurality of wireless access points. The methodincludes: upon receiving a communication switching instruction from afirst wireless access point used for communication with the imageprocessing apparatus, disconnecting communication with the firstwireless access point and transmitting, to a second wireless accesspoint, a link request to establish communication with the secondwireless access point of a new communication destination included in theswitching instruction; and displaying, on the display unit, the capturedimage continuously acquired from the image capturing unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

According to still another aspect of the present invention, there isprovided a method of controlling an image display apparatus including animage capturing unit which captures an image of physical space, adisplay unit which receives, from an image processing apparatus, acomposite image of a virtual image and the captured image captured bythe image capturing unit, which is composited by the image processingapparatus, and displays the composite image, an image superimpositiondisplay unit which superimposes a message image on a display image ofthe display unit, and a wireless communication unit which communicateswith the image processing apparatus via one of a plurality of wirelessaccess points. The method includes: upon receiving a communicationswitching instruction from a first wireless access point used forcommunication with the image processing apparatus, disconnectingcommunication with the first wireless access point and transmitting, toa second wireless access point, a link request to establishcommunication with the second wireless access point of a newcommunication destination included in the switching instruction; andcausing the image superimposition display unit to superimpose themessage image on the display image of the display unit until switchingfrom the first wireless access point to the second wireless access pointfinishes as communication destination switching.

According to still another aspect of the present invention, there isprovided a method of controlling an image display apparatus including animage capturing unit which captures an image of physical space, astorage unit which stores a virtual image, a composition unit whichcomposites the captured image captured by the image capturing unit andthe virtual image stored in the storage unit, a display unit whichdisplays a composite image composited by the composition unit, and awireless communication unit which communicates with an image processingapparatus via one of a plurality of wireless access points. The methodincludes: upon receiving a communication switching instruction from afirst wireless access point used for communication with the imageprocessing apparatus, disconnecting communication with the firstwireless access point and transmitting, to a second wireless accesspoint, a link request to establish communication with the secondwireless access point of a new communication destination included in theswitching instruction; and displaying, on the display unit, thecomposite image composited by the composition unit using the capturedimage and the virtual image received from the image processing apparatusimmediately before the switching and stored in the storage unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

According to another aspect of the present invention, acomputer-readable medium has stored thereon a computer program to causea computer to execute control of an image display apparatus including animage capturing unit which captures an image of physical space, adisplay unit which receives, from an image processing apparatus, acomposite image of a virtual image and the captured image captured bythe image capturing unit, which is composited by the image processingapparatus, and displays the composite image, and a wirelesscommunication unit which communicates with the image processingapparatus via one of a plurality of wireless access points, by causingthe computer to execute: a communication control step of, upon receivinga communication switching instruction from a first wireless access pointused for communication with the image processing apparatus,disconnecting communication with the first wireless access point andtransmitting, to a second wireless access point, a link request toestablish communication with the second wireless access point of a newcommunication destination included in the switching instruction; and adisplay control step of displaying, on the display unit, the capturedimage continuously acquired from the image capturing unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

According to yet another aspect of the present invention, acomputer-readable medium has stored thereon a computer program to causea computer to execute control of an image display apparatus including animage capturing unit which captures an image of physical space, adisplay unit which receives, from an image processing apparatus, acomposite image of a virtual image and the captured image captured bythe image capturing unit, which is composited by the image processingapparatus, and displays the composite image, an image superimpositiondisplay unit which superimposes a message image on a display image ofthe display unit, and a wireless communication unit which communicateswith the image processing apparatus via one of a plurality of wirelessaccess points, by causing the computer to execute: a communicationcontrol step of, upon receiving a communication switching instructionfrom a first wireless access point used for communication with the imageprocessing apparatus, disconnecting communication with the firstwireless access point and transmitting, to a second wireless accesspoint, a link request to establish communication with the secondwireless access point of a new communication destination included in theswitching instruction; and a display control step of causing the imagesuperimposition display unit to superimpose the message image on thedisplay image of the display unit until switching from the firstwireless access point to the second wireless access point finishes ascommunication destination switching.

According to still another aspect of the present invention, acomputer-readable medium has stored thereon a computer program to causea computer to execute control of an image display apparatus including animage capturing unit which captures an image of physical space, astorage unit which stores a virtual image, a composition unit whichcomposites the captured image captured by the image capturing unit andthe virtual image stored in the storage unit, a display unit whichdisplays a composite image composited by the composition unit, and awireless communication unit which communicates with an image processingapparatus via one of a plurality of wireless access points, by causingthe computer to execute: a communication control step of, upon receivinga communication switching instruction from a first wireless access pointused for communication with the image processing apparatus,disconnecting communication with the first wireless access point andtransmitting, to a second wireless access point, a link request toestablish communication with the second wireless access point of a newcommunication destination included in the switching instruction; and adisplay control step of displaying, on the display unit, the compositeimage composited by the composition unit using the captured image andthe virtual image received from the image processing apparatusimmediately before the switching and stored in the storage unit untilswitching from the first wireless access point to the second wirelessaccess point finishes as communication destination switching.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an image display systemaccording to a first embodiment of the present invention;

FIGS. 2A and 2B are sequence charts showing the data flow between theapparatuses and their process contents according to the first embodimentof the present invention;

FIG. 3 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the firstembodiment of the present invention;

FIG. 4 is a functional block diagram of an image display apparatusaccording to a second embodiment of the present invention;

FIG. 5A is a view showing an example of a message image displayed by theOn Screen Display (OSD) unit of the image display apparatus according tothe second embodiment of the present invention;

FIG. 5B is a view showing another example of the message image displayedby the OSD unit of the image display apparatus according to the secondembodiment of the present invention;

FIG. 6 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the secondembodiment of the present invention;

FIG. 7 is a functional block diagram of an image display systemaccording to a third embodiment of the present invention;

FIG. 8 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the thirdembodiment of the present invention;

FIG. 9 is a flowchart illustrating details of CG correction processingin step S806 according to the third embodiment of the present invention;

FIG. 10 is a functional block diagram of a conventional videosee-through mixed reality wireless system; and

FIG. 11 is a view showing the concept of position and orientationinformation generation using a marker in the prior art.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will now be described indetail with reference to the drawings. It should be noted that therelative arrangement of the components, the numerical expressions andnumerical values set forth in these embodiments do not limit the scopeof the present invention unless it is specifically stated otherwise.

FIG. 1 is a functional block diagram of an image display systemaccording to a first exemplary embodiment of the present invention.

An image display apparatus 101 is assumed to be, for example, a videosee-through HMD. The image display apparatus 101 has an image capturingunit 104, three-dimensional position and orientation measurement sensor105, wireless communication I/F (wireless communication unit) 106,display control unit 107, frame buffer (storage unit) 108, and displayunit 109.

The image capturing unit 104, three-dimensional position and orientationmeasurement sensor 105, wireless communication I/F 106, and display unit109 correspond to the image capturing unit 1004, three-dimensionalposition and orientation measurement sensor 1005, wireless communicationI/F 1006, and display unit 1007 in FIG. 10, respectively. Wireless APs102 a and 102 b (first wireless access point and second wireless accesspoint) correspond to the wireless access points 1002 a and 1002 b inFIG. 10, respectively. Hence, FIG. 1 will be described concerningdifferent points from FIG. 10.

The three-dimensional position and orientation measurement sensor 105outputs information to grasp the motion or the position and orientationof the HMD user's head. The three-dimensional position and orientationmeasurement sensor 105 is incorporated in the image display apparatus101. However, this is not essential when using an application thatallows for grasping the position and orientation based on a marker in acaptured image.

The wireless communication I/F 106 transmits a captured image (physicalspace image) and position and orientation information to the wirelessaccess point (AP) 102 a or 102 b and receives a composite imagegenerated by an image processing apparatus 103 via the wireless AP 102 aor 102 b. The wireless communication I/F 106 performs the followingcommunication control for each wireless access point. Upon receiving acommunication disconnection instruction from the first wireless accesspoint which is being used for communication with the image processingapparatus 103, the wireless communication I/F 106 disconnects thecommunication with the first wireless access point. Then, the wirelesscommunication I/F 106 transmits a link request to the second wirelessaccess point to establish communication with the second wireless accesspoint that is the new communication destination contained in thedisconnection instruction.

The wireless communication I/F 106 needs to perform real-time processingand preferably uses a communication standard capable of high-bandwidthtransmission. For example, high-speed wireless communication based onIEEE 802.11 wireless LAN or IEEE 802.15 wireless PAN standard is used.This embodiment will assume the use of UWB (Ultra Wide Band). Thetransmission rate of UWB is several hundred Mbps.

The display control unit 107 controls display upon handover as wirelessAP switching processing. This processing will be described later indetail. The frame buffer 108 is a memory which stores each frame of areceived MR image. How to use the stored MR image will also be describedlater.

The image processing apparatus 103 renders a CG image based on thecaptured image and the position and orientation information receivedfrom the image display apparatus 101 and executes composition processingof generating a composite image of the CG image and the captured image.

The image processing apparatus 103 has a communication I/F(communication unit) 110, position and orientation informationgeneration unit 111, CG rendering unit 112, content storage unit 113,image composition unit 114, switching determination/instruction unit115, and wireless AP position information storage unit 116.

The communication I/F 110 performs data communication with the wirelessAP 102 a or 102 b. The communication I/F 110 is used to transmit acaptured image and position and orientation information received by thewireless AP 102 a or 102 b and a composite image generated by the imageprocessing apparatus 103. The communication I/F 110 may be a wirelesscommunication I/F capable of direct wireless communication with thewireless AP 102 a or 102 b without intervening a LAN 117.

The position and orientation information generation unit 111 generatesthe position and orientation information of the HMD user based on thereceived captured image. In particular, the position and orientationinformation generation unit 111 generates position and orientationinformation by extracting a marker or a feature point serving as amarker from the captured image. The position and orientation informationgeneration unit 111 can increase the accuracy by complementarily using acaptured image from an objective viewpoint (not shown) or informationobtained by the three-dimensional position and orientation measurementsensor 105 of the image display apparatus 101, thus coping with a casethat the captured image contains neither a marker nor a feature point tobe used.

The CG rendering unit (virtual image generation unit) 112 renders a CGimage that is a virtual image based on the position and orientationinformation and a content acquired from the content storage unit 113.The content storage unit 113 stores contents for generating a CG image.The image composition unit 114 composites the virtual image generated bythe CG rendering unit 112 with the background, that is, the capturedimage captured in the image display apparatus 101.

Based on the positional relationship between the image display apparatus101 and the plurality of wireless APs 102 a and 102 b, the switchingdetermination/instruction unit 115 calculates the distances between theapparatuses. The switching determination/instruction unit 115 determinesbased on the calculation result whether it is necessary to leave thecommunication area of a wireless AP 102 and switch to the communicationarea of another wireless AP 102. Upon determining that switching isnecessary, the switching determination/instruction unit 115 transmits aswitching instruction of the wireless AP 102 to the wireless AP 102 ofthe current link, the image display apparatus 101, and the wireless APof the new link establishment target.

The wireless AP position information storage unit 116 is a data tablewhich stores the position information of each wireless AP 102. Thewireless AP position information storage unit stores the information ofthe installation position and can be updated.

The LAN (Local Area Network) 117 connects the plurality of wireless APs102 and the image processing apparatus 103. In the first embodiment, twowireless APs 102 a and 102 b and one image processing apparatus 103 areconnected. However, the number of apparatuses changes depending on themoving range of the HMD user and the area coverage of each wireless AP102. If the process load is too heavy for one image processingapparatus, a plurality of image processing apparatuses may be connectedto the LAN 117 to distribute the process load.

The description of the first embodiment will be made assuming that thefunctional blocks of the image display apparatus 101 are included in oneapparatus. However, the same functions can also be implemented byseparately including the functions in two or more apparatuses.

The image display apparatus 101 transmits a captured image and theposition and orientation information of the three-dimensional positionand orientation measurement sensor 105 together to the wireless AP 102 aof the current link via the wireless communication I/F 106.

The wireless AP 102 a transmits the received captured image and positionand orientation information to the image processing apparatus 103 viathe LAN 117.

The image processing apparatus 103 grasps the position and orientationof the image display apparatus 101 using the received captured image andposition and orientation information and determines the line-of-sightposition of the HMD user. Based on the obtained position and orientationinformation and content for CG rendering, the image processing apparatusgenerates a desired CG image and superimposes it on the receivedcaptured image. The obtained composite image is transmitted to thewireless AP 102 a via the communication I/F 110.

Then, the wireless AP 102 a transmits the received composite image tothe image display apparatus 101 by wireless communication.

If the composite image is normally received, the image display apparatus101 displays the composite image for the HMD user via the display unit109. General image formation processing has been described above.

An operation in switching will be described next.

The HMD user who moves in a wireless area may go out of thecommunication area of the wireless AP 102 a of the current link. Whetherto go out of the communication area is determined based on distanceinformation calculated based on the position information of the wirelessAP 102 stored in the wireless AP position information storage unit 116and the position information of the image display apparatus 101.

Whether it is necessary to switch from the wireless AP 102 a of thecurrent link to the wireless AP 102 b is determined based on thedistance to each wireless AP. Prediction based on a time-rate change mayalso be taken into consideration. For example, if it can be determinedthat the user is leaving the wireless AP 102 a of the current link andapproaching the wireless AP 102 b, the switching instruction is sent toboth the wireless AP 102 of the current link and that of the next link.The instruction is also transmitted to the image display apparatus 101via the wireless AP 102 a.

The image display apparatus 101 that has received the instruction cannotreceive the composite image during handover (until the end ofswitching). During this time, the image display apparatus displays thecaptured image acquired in real time. Even after establishment of thelink, the image display apparatus continuously displays the capturedimage until reception of a new composite image. This processing andother display control will be described later in detail.

The above-described arrangement and processing enable to ensure the HMDuser's view and image display without impairing the mixed reality evenwhen composite image transmission is interrupted during handover.

The data flow between the image display apparatus 101, wireless APs 102a and 102 b, and the image processing apparatus 103 and their processcontents will be described next with reference to FIGS. 2A and 2B.

FIGS. 2A and 2B are sequence charts showing the data flow between theapparatuses and their process contents according to the first embodimentof the present invention.

FIGS. 2A and 2B particularly show the sequence of processing ofswitching the link from the wireless AP 102 a (to be referred as an AP 1hereinafter) of current wireless communication to the wireless AP 102 b(to be referred as an AP 2 hereinafter) as the HMD user moves.

In step S201, the image display apparatus 101 transmits an acquiredcaptured image and the sensor information of the three-dimensionalposition and orientation measurement sensor to the AP 1 in the linkedstate. Note that the image display apparatus 101 periodically acquires acaptured image and sensor information and therefore transmits these datato the AP 1 asynchronous to the processes of the remaining apparatuses(steps S201 a to S201 d).

In step S202, the AP 1 transmits the received captured image and sensorinformation to the image processing apparatus 103.

In step S203, the image processing apparatus 103 generates position andorientation information by extracting a feature point from the receivedcaptured image. The position and orientation information can begenerated either by calculating the position and orientation based onthe sensor information or by complementarily using the sensorinformation. In step S204, the image processing apparatus 103 generatesa CG image as a virtual image based on the generated position andorientation information. In step S205, the image processing apparatus103 generates a mixed reality image (MR image) as a composite image bysuperimposing the generated CG image on the received captured image. Instep S206, the image processing apparatus 103 transmits the generatedcomposite image to the AP 1.

In step S207, the AP 1 transmits the received composite image to theimage display apparatus 101. The image display apparatus 101 displaysthe received composite image.

In step S208, the image processing apparatus 103 calculates thedistances between the apparatuses based on the position of the imagedisplay apparatus 101 and the positions of the APs 1 and 2 anddetermines whether AP switching is necessary. In this example, assumethat the HMD user who is present in the areas of the APs 1 and 2 isexpected to leave the area of the AP 1 of the current link and enter thearea of the AP 2. In this case, the image processing apparatus 103determines switching from the AP 1 to the AP 2.

In step S209, the image processing apparatus 103 transmits a linkswitching instruction (disconnection) to the AP 1 of the current link inaccordance with the AP switching determination result. The transmittedinformation includes the switching instruction, the physical ID of thewireless AP of the next link destination, and its channel information inuse. The channel number in use is not indispensable. However, if it isgrasped in advance, the time until link establishment can be shortened.

In step S210, the AP 1 transmits the switching instruction from theimage processing apparatus 103 to the image display apparatus 101. Theimage display apparatus 101 continuously displays the composite imageuntil receiving a composite image corresponding to the previouslytransmitted captured image (from step S201 b). The contents of thedisplay control processing on the side of the image display apparatus101 will be described later with reference to another flowchart.

In step S211, the image processing apparatus 103 transmits a switchinginstruction (link) to the AP 2 of the new link establishment target.Link establishment is not actually done based on this instruction.However, it is possible to shorten the time until link establishment bynotifying the AP 2 of the physical ID and the like of the image displayapparatus 101 as the communication destination.

In step S206 d, the image processing apparatus 103 transmits, to the AP1, a composite image corresponding to the final captured image receivedfrom the image display apparatus 101 via the AP 1.

In step S207 d, the AP 1 transmits the received composite image to theimage display apparatus 101.

In step S212, the image display apparatus 101 disconnects the link tothe AP 1 and transmits a link request to the AP 2 of the new linkestablishment target.

In step S213, the AP 2 receives the link request from the image displayapparatus 101, establishes a link, and notifies the image displayapparatus 101 of it.

In step S214, the image display apparatus 101 performs display controlbefore updating display of the final composite image received via the AP1 in step S207 d to change the display image until handover finishes andan image from the AP 2 of the new link is received. The display controlwill be described later in detail.

In step S215, the image display apparatus 101 transmits the acquiredcaptured image and sensor information to the AP 2.

In step S216, the AP 2 transmits the received captured image and sensorinformation to the image processing apparatus 103.

In step S217, the image processing apparatus 103 generates position andorientation information by extracting a feature point from the receivedcaptured image. The contents are the same as in step S203. In step S218,the image processing apparatus 103 generates a CG image as a virtualimage based on the generated position and orientation information. Thecontents are the same as in step S204. In step S219, the imageprocessing apparatus 103 generates a mixed reality image (MR image) as acomposite image by superimposing the generated CG image on the receivedcaptured image. The contents are the same as in step S205. In step S220,the image processing apparatus 103 transmits the generated compositeimage to the AP 2.

In step S221, the AP 2 transmits the received composite image to theimage display apparatus 101.

Upon receiving the composite image to be displayed, the image displayapparatus 101 finishes the display image switching control and displaysthe received composite image in step S222.

The main sequence of wireless AP switching has been described above.

The display image switching processing of the image display apparatus101 in step S214 of FIG. 2B will be described next with reference toFIG. 3.

FIG. 3 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the firstembodiment of the present invention.

In step S301, the image display apparatus 101 determines whether aswitching instruction from the image processing apparatus 103 isreceived via the wireless AP. If a switching instruction is received(YES in step S301), the process advances to step S302. If no switchinginstruction is received (NO in step S301), the process advances to stepS314. Note that the switching instruction includes connectiondestination information representing the wireless AP to be connectednext.

In step S302, the image display apparatus 101 determines whether allcomposite images corresponding to captured images already transmitteduntil reception of the switching instruction are received. If allcomposite images are received (YES in step S302), the process advancesto step S303. If a composite image is not received yet (NO in stepS302), the process waits until all composite images are received.

When all composite images are received, the image display apparatus 101starts switching control in step S303. More specifically, the imagedisplay apparatus disconnects the link to the wireless AP of the currentlink and starts negotiating with the wireless AP of the new linkestablishment target.

In step S304, the image display apparatus 101 determines whether todisplay a composite image acquired immediately before the linkdisconnection until new link establishment and subsequent reception of adisplay image (composite image).

This determination is done based on whether, for example, the compositeimage contains a CG image and whether the HMD user has moved largely (apredetermined distance or more) at the time of link disconnection. Inexemplary embodiments, if the composite image contains no CG image, thecaptured image is directly displayed and display of the composite imageis not selected. During movement, the physical world to be displayed forthe HMD user may be widely different from the composite image. Hence,display of the composite image is not selected in this case.

If it is determined to display the composite image (YES in step S304),the process advances to step S305. To directly display the capturedimage without displaying the composite image (NO in step S304), theprocess advances to step S306. In selecting the image to be displayed,whether to display the composite image can be determined every displayimage update timing. Alternatively, determination may be done at thestart of handover so that the selected image is continuously displayeduntil the end of display switching.

In step S305, the image display apparatus 101 reads out the compositeimage acquired and stored in the frame buffer 108 immediately before thelink disconnection.

Next, in step S307, the image display apparatus 101 calculates thecurrent position and orientation information. More specifically, theimage display apparatus calculates the position and orientationinformation of the HMD user's head based on the output information ofthe three-dimensional position and orientation measurement sensor.

In step S308, the image display apparatus 101 determines whether toexecute composite image correction processing. The composite imagecorrection processing is done in accordance with a change of theposition and orientation of the HMD user. To execute correctionprocessing (YES in step S308), the process advances to step S309. Ifcorrection processing is not to be executed (NO in step S308), theprocess advances to step S310.

In step S309, in moving in the left, right, upper, or lower direction,the image display apparatus 101 executes correction processing ofgenerating an image obtained by moving the composite image in thedesired direction. In moving back and forth, the resolution of thecomposite image is converted. If movement is done in a plurality ofdirections, the image display apparatus executes correction processingbased on the combination of the directions. Then processing advances tostep S310.

In step S306 (if it has been determined in step S304 that a compositeimage is not to be displayed), the image display apparatus 101 acquiresa captured image. Then, processing advances to step S310.

In step S310, the image display apparatus 101 displays one of thecorrected composite image, uncorrected composite image, and capturedimage on the display unit 109 while updating until establishment of thelink with the new wireless AP and reception of a composite image.

In step S311, the image display apparatus 101 determines whether the newlink has been established. If the link has been established, andcaptured image transmission has started (YES in step S311), the processadvances to step S312. If the link has not been established yet, andhandover is progressing (NO in step S311), the process returns to stepS304.

In step S312, the image display apparatus 101 determines whetherhandover has finished, and a composite image corresponding to thecaptured image transmitted to the wireless AP of the newly establishedlink has been received. This is because there is a time lag fromacquisition and transmission of a captured image to generation andreception of a composite image. If a composite image has been receivedfrom the wireless AP of the newly established link (YES in step S312),the process advances to step S313. If no composite image has beenreceived (NO in step S312), the process returns to step S304.

When the composite image to be displayed is received, the displaycontrol unit 107 of the image display apparatus 101 switches the displayimage in step S313. More specifically, display control of one of thecorrected composite image, uncorrected composite image, and capturedimage, which is displayed so far, is ended.

In step S314, the image display apparatus 101 displays the receivedcomposite image on the display unit 109. After that, the image displayapparatus displays, on the display unit 109, composite images that arecontinuously received until switching of the wireless AP occurs again.

As described above, according to the first embodiment, a captured imagecaptured in real time is displayed at the time of switching the wirelessAP, thus ensuring the HMD user's view during handover untilestablishment of the link with the next wireless AP. It is also possibleto continue display without impairing the mixed reality by displayingthe composite image immediately before link disconnection whilecorrecting it in accordance with the HMD's position and orientation.

In the first embodiment, an arrangement has been described, whichcontrols display of captured images continuously acquired in real timeuntil establishment of the link with the next wireless AP and receptionof a composite image. In a second embodiment, an arrangement will bedescribed, which presents a superimposed message to call the HMD user'sattention while performing the same display control.

The functional arrangement of an image display apparatus according tothe second embodiment will be described first.

FIG. 4 is a functional block diagram of the image display apparatusaccording to the second embodiment of the present invention.

An image display apparatus 401 is assumed to be, for example, a videosee-through HMD. The image display apparatus 401 has an image capturingunit 404, three-dimensional position and orientation measurement sensor405, wireless communication I/F 406, display control unit 407, framebuffer 408, display unit 409, and OSD unit 410.

The image capturing unit 404, three-dimensional position and orientationmeasurement sensor 405, and wireless communication I/F 406 correspond tothe image capturing unit 104, three-dimensional position and orientationmeasurement sensor 105, and wireless communication I/F 106 in FIG. 1,respectively. The display control unit 407, frame buffer 408, anddisplay unit 409 correspond to the display control unit 107, framebuffer 108, and display unit 109 in FIG. 1, respectively. Hence, adetailed description of them will not be repeated, and the OSD unit 410as a new component will be described. The image display apparatus 401,and the image processing apparatus 103 and the wireless APs 102 a and102 b in FIG. 1 implement an image display system.

The OSD unit 410 is an OSD (On Screen Display) unit (imagesuperimposition display unit) which superimposes a message image formedfrom, for example, a text or an image on a display image. An example ofdisplay contents and the process sequence will be described later.Particularly, if handover occurs when the HMD user is moving a longdistance, the OSD unit 410 allows displaying a message for the purposeof calling the HMD user's attention until wireless AP switching finishesand composite image reception is enabled.

An example of display by the OSD unit 410 will be described withreference to FIGS. 5A and 5B.

FIGS. 5A and 5B are views showing examples of message images displayedby the OSD unit of the image display apparatus according to the secondembodiment of the present invention.

FIG. 5A shows an example of a message representing that wireless APswitching is progressing. The message is actually continuously displayedafter link establishment to transmission of a captured image andreception of a composite image corresponding to the transmitted capturedimage. The display image serving as the background is the capturedimage, composite image, or corrected composite image, as in the firstembodiment.

When the image displayed for the HMD user contains a CG image, itabruptly disappears upon switching the display image to the capturedimage. The composite image is displayed after predetermined processingand therefore has a delay with respect to the captured image. The secondembodiment can produce a sense of security for the user by presentingthe cause of the sense of incongruity generated upon switching.

FIG. 5B shows a display example in which a message to call the user'sattention is added to the message representing that switching isprogressing. It is possible to ensure safety for the user by limitingarea movement until the display image switching processing finishes.

The display image switching processing of the image display apparatus401 according to the second embodiment will be described next withreference to FIG. 6.

FIG. 6 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the secondembodiment of the present invention.

Steps S601 to S609 and steps S610 to S614 in FIG. 6 correspond to stepsS301 to S309 and steps S310 to S314 in FIG. 3 of the first embodiment,and a detailed description thereof will not be repeated.

Referring to FIG. 6, in step S609 a, the image display apparatus 401determines whether to execute warning display processing of displaying amessage to call the HMD user's attention. Switching and selection of themessage display can be done by the HMD user either from an input unitsuch as a switch provided on the image display apparatus 401 or from theinput unit of the image processing apparatus 103. To execute warningdisplay processing (YES in step S609 a), the process advances to stepS609 b. If warning display processing is not to be executed (NO in stepS609 a), the process advances to step S610.

In step S609 b, the image display apparatus 401 executes the warningdisplay processing. More specifically, the OSD unit 410 superimposes amessage prepared in advance to call the user's attention on the displayimage.

As described above, according to the second embodiment, in switching thewireless AP and displaying a captured image captured in real time or acomposite image acquired immediately before disconnection, a message ispresented to the user, thus limiting his/her action or calling his/herattention. An example of the message represents that handover isprogressing.

In the first and second embodiments, an arrangement has been described,in which the image display apparatus transmits a captured image to theimage processing apparatus and receives and displays a composite imagecontaining a CG image which is superimposed in the image processingapparatus. In a third embodiment, an arrangement will be described, inwhich an image display apparatus transmits at least position andorientation information to an image processing apparatus, receives a CGimage generated based on the position and orientation information in theimage processing apparatus, and composites the captured image with theCG image.

FIG. 7 is a functional block diagram of the image display apparatusaccording to the third embodiment of the present invention.

An image display apparatus 701 is assumed to be, for example, a videosee-through HMD. The image display apparatus 701 has an image capturingunit 704, three-dimensional position and orientation measurement sensor705, wireless communication I/F 706, frame buffers 707 and 708, displayunit 709, and OSD unit 710. The image display apparatus 701 alsoincludes a correction processing unit 711, correction processingdetermination unit 712, selector 713, image composition unit 714, andposition and orientation information generation unit 715.

The image capturing unit 704, three-dimensional position and orientationmeasurement sensor 705, and wireless communication I/F 706 correspond tothe image capturing unit 404, three-dimensional position and orientationmeasurement sensor 405, and wireless communication I/F 406 in FIG. 4,respectively. The display unit 709 and the OSD unit 710 correspond tothe display unit 409 and the OSD unit 410 in FIG. 4, respectively.Hence, a detailed description of them will not be repeated, and newcomponents will be described.

Wireless APs 702 a and 702 b and a LAN 7117 correspond to the wirelessAPs 102 a and 102 b and the LAN 117 in FIG. 1, respectively.

In the image display apparatus 701, the wireless communication I/F 706transmits position and orientation information to the wireless AP 702 aor 702 b and receives a CG image generated by an image processingapparatus 703 via the wireless AP 702 a or 702 b. The image displayapparatus 701 transmits position and orientation information here.However, it may transmit a captured image together.

The frame buffer 707 is a memory which stores each frame of an acquiredcaptured image. The frame buffer 708 is a memory which stores each frameof a CG image received from the image processing apparatus 703.

The correction processing unit 711 corrects the display position of theCG image. This will be described later in detail. The correctionprocessing determination unit 712 determines based on the position andorientation information whether correction of the display position withrespect to the virtual image is possible, and if correction is to beperformed, determines which kind of correction is to be executed. Thiswill be described later in detail.

The selector 713 selects a CG image to be composited with the capturedimage. Normally, the selector selects the received CG image. Duringswitching processing for handover, the selector selects an alreadyacquired CG image stored in the frame buffer 708 or a CG image afterdisplay position correction.

The image composition unit 714 composites the captured image with the CGimage. The image composition unit 714 can composite a captured imagetemporarily stored in the frame buffer 707 with a CG image correspondingto the captured image. The image composition unit 714 can also compositea captured image continuously acquired in real time with a CG imagegenerated at a display position predicted in consideration of the timingof composition.

The position and orientation information generation unit 715 generatesthe HMD user's position and orientation information based on thecaptured image and the output information from the three-dimensionalposition and orientation measurement sensor 705.

The image processing apparatus 703 renders a CG image based on theposition and orientation information received from the image displayapparatus 701.

The image processing apparatus 703 comprises a communication I/F 7110,CG rendering unit 7112, content storage unit 7113, switchingdetermination/instruction unit 7115, wireless AP position informationstorage unit 7116, and position correction unit 717.

The communication I/F 7110, CG rendering unit 7112, and content storageunit 7113 correspond to the communication I/F 110, CG rendering unit112, and content storage unit 113 in FIG. 1, respectively. The switchingdetermination/instruction unit 7115 and the wireless AP positioninformation storage unit 7116 correspond to thedetermination/instruction unit 115 and the wireless AP positioninformation storage unit 116 in FIG. 1, respectively. Hence, a detaileddescription of them will not be repeated, and new components will bedescribed.

The position correction unit 717 corrects the position and orientationinformation generated by the image display apparatus 701 in accordancewith the image generation timing. For example, time-serially graspedposition and orientation information in the past is corrected toposition and orientation information at the time of image composition.The processing can be done using a general prediction method. A methodusing a Kalman filter is well-known. Correction is done based on theimage composition timing here. However, it may be done based on thecomposite image display timing. In this case, for the captured image, ashort time lag is generated from acquisition to display. However, the CGimage appears to ensure real time processing unless the there is asignificant error in the prediction.

Processing without CG image correction processing will be describednext.

The captured image obtained by the image capturing unit 704 is stored inthe frame buffer 707. The captured image is used by the position andorientation information generation unit 715 to calculate the positionand orientation information of the image display apparatus 701 togetherwith the output from the three-dimensional position and orientationmeasurement sensor 705.

The generated position and orientation information is transmitted fromthe wireless communication I/F 706 to the image processing apparatus 703via the wireless AP 702 a. In the image processing apparatus 703, the CGrendering unit 7112 renders a CG image based on the received positionand orientation information. The CG image is transmitted from thecommunication I/F 7110 to the image display apparatus 701 via thewireless AP 702 a. When image composition is done by chromakeyprocessing, a specific color (chromakey color) which is not used for CGrendering but is to be used for chromakey composition is also sent asinformation. If specific color designation is not performed for eachframe, negotiation with the image display apparatus 701 is done inadvance to use the same specific color during a predetermined period.

In the image display apparatus 701, the image composition unit 714chromakey-composites the received CG image with the captured imagestored in the frame buffer 707 based on the specific color. The capturedimage to be associated with the CG image is the image used to acquirethe position and orientation information used for generation of the CGimage. For image composition using an α channel (semitransparent imagecomposition), the image processing apparatus 703 transmits an α channelthat is two-dimensional data of α values (mat values) of the pixelstogether with the CG image. The display unit 709 displays the obtainedcomposite image, that is, MR image to produce mixed reality for theuser.

This arrangement obviates captured image transmission from the imagedisplay apparatus 701 to the wireless AP 702 a. For this reason, if, forexample, the resolution, tone expression, and frame rate of the capturedimage are the same as those of the CG image, the amount of data to betransmitted is almost ½ as compared to the general MR system arrangementdescribed in the prior art. The actual data amount is larger than ½because the chromakey color or α channel is transmitted together withthe CG image.

The captured image to be employed in composition is the captured imageused to calculate the position and orientation information measured forCG image generation. It is therefore possible to provide an MR imagewithout any time lag between the captured image and the CG image and anysense of incongruity. In addition, when the image processing apparatus703 generates the CG image after predicting the display position inaccordance with the composition timing, the time lag between the CGimage and the physical world can be reduced.

The above-described arrangement and processing obviate captured imagetransmission for MR image display and enable maintaining of the samedisplay quality.

The display image switching processing of the image display apparatus701 will be described next with reference to FIG. 8.

FIG. 8 is a flowchart illustrating the display image switchingprocessing of the image display apparatus according to the thirdembodiment of the present invention.

In step S801, the image display apparatus 701 determines whether aswitching instruction determined by the image processing apparatus 703is received via the wireless AP. If a switching instruction is received(YES in step S801), the process advances to step S802. If no switchinginstruction is received (NO in step S801), the process advances to stepS814.

In step S802, the image display apparatus 701 determines whether all CGimages corresponding to position and orientation information alreadytransmitted until reception of the switching instruction are received.If all CG images are received (YES in step S802), the process advancesto step S803. If a CG image is not received yet (NO in step S802), theprocess waits until all CG images are received.

When all CG images are received, the image display apparatus 701 startsswitching control in step S803.

In step S804, the image display apparatus 701 reads out a CG imageacquired immediately before the link disconnection and stored in theframe buffer 708.

In step S805, the image display apparatus 701 determines whether toexecute CG image correction processing (CG correction processing). Toexecute CG correction processing (YES in step S805), the processadvances to step S806. If CG correction processing is not to be executed(NO in step S805), the process advances to step S807.

In step S806, the image display apparatus 701 executes CG correctionprocessing. Details of the correction processing will be described laterwith reference to a flowchart shown in FIG. 9. Processing then advancesto step S809.

In step S807, the image display apparatus 701 determines whether toexecute warning display processing of displaying a message to call theHMD user's attention. To execute warning display processing (YES in stepS807), the process advances to step S808. If warning display processingis not to be executed (NO in step S807), the process advances to stepS809.

In step S808, the image display apparatus 701 executes the warningdisplay processing. This processing is the same as in step S609 b inFIG. 6 of the second embodiment.

In step S809, the image display apparatus 701 composites the CG imagewith the captured image. In step S810, the image display apparatus 701displays the generated composite image on the display unit 709 whileupdating until establishment of the link with the new wireless AP.

In step S811, the image display apparatus 701 determines whether the newlink has been established. If the link has been established, andposition and orientation information transmission has started (YES instep S811), the process advances to step S812. If the link has not beenestablished yet, and handover is progressing (NO in step S811), theprocess returns to step S804.

In step S812, the image display apparatus 701 determines whetherhandover has finished, and a CG image corresponding to the position andorientation information transmitted to the wireless AP of the newlyestablished link has been received. If a CG image has been received fromthe wireless AP of the newly established link (YES in step S812), theprocess advances to step S813. If no CG image has been received (NO instep S812), the process returns to step S804.

In step S813, the image display apparatus 701 composites the received CGimage with the captured image. In step S814, the image display apparatus701 displays the generated composite image on the display unit 709.Processing then returns to step S801.

The CG correction processing in step S806 according to the thirdembodiment of the present invention will be described next withreference to FIG. 9.

FIG. 9 is a flowchart illustrating details of CG correction processingin step S806 according to the third embodiment of the present invention.

In step S901, the image display apparatus 701 calculates the position ofa CG image acquired immediately before the link disconnection and storedin the frame buffer 708. In step S902, the image display apparatus 701generates current position and orientation information based on sensorinformation from the three-dimensional position and orientationmeasurement sensor 705.

In step S903, the image display apparatus 701 grasps in which directionthe HMD user is presently moving or directing the line-of-sight positionand determines whether the movement direction is the left, right, upper,or lower direction. If it is a movement in the left, right, upper, orlower direction or a similar movement of the line-of-sight position (YESin step S903), the process advances to step S904. Otherwise (NO in stepS903), the process advances to step S905.

In step S904, the image display apparatus 701 determines whether the CGimage is movable. More specifically, the image display apparatuscalculates the moving amount up to the current position and determineswhether the display position of the CG image can be shifted inaccordance with the moving amount without any inconvenience. The imagedisplay apparatus determines that the CG image is movable when an imageexcept the acquired CG image can be displayed in the existing CG imagedisplay region or an image behind or on the left or right side of theobtained CG image is required as a result of a motion such as rotation.If the CG image is movable (YES in step S904), the process advances tostep S906. If the CG image is not movable (NO in step S904), the processadvances to step S907.

In step S905, the image display apparatus 701 determines whether the HMDuser is moving back and forth. If the user is moving back and forth (YESin step S905), the process advances to step S908. If the user is notmoving back and forth (NO in step S905), the process advances to stepS907.

When movement in the upper, lower, left, and right directions ispossible, the image display apparatus 701 translates the CG image instep S906. Processing then advances to step S909.

When the CG image is not movable, the image display apparatus 701displays a warning message on the display unit 709 in step S907.Processing then advances to step S909.

When movement in the back and forth directions is possible, the imagedisplay apparatus 701 enlarges or reduces the CG image in step S908.Particularly, in moving toward the CG image rendering position, anenlargement process is executed. In moving away from the CG image, areduction process is executed. If motions in the upper, lower, left, andright directions and movements in the back and forth directions arecombined, appropriate processes are executed, although they will not beexplained here. Processing then advances to step S909.

In step S909, the image display apparatus 701 outputs the CG image withthe corrected position. Processing then returns to step S809 of FIG. 8.

As described above, according to the third embodiment, even when theimage display apparatus composites a captured image and a CG image, itis possible to composite the captured image acquired in real time withthe CG image according to the position and orientation of that time anddisplay the composite image unless the movement is large. If CG imagecorrection is impossible, the captured image may be displayed, as in thefirst embodiment. Alternatively, a warning message may be presented tothe user, as in the second embodiment.

Note that the present invention can be applied to an apparatuscomprising a single device or to a system constituted by a plurality ofdevices.

Furthermore, the invention can be implemented by supplying a softwareprogram, which implements the functions of the foregoing embodiments,directly or indirectly to a system or apparatus, reading the suppliedprogram code with a computer of the system or apparatus, and thenexecuting the program code. In this case, so long as the system orapparatus has the functions of the program, the mode of implementationneed not rely upon a program.

In this case, so long as the system or apparatus has the functions ofthe program, the program may be executed in any form, such as objectcode, a program executed by an interpreter, or script data supplied toan operating system.

Example of storage media that can be used for supplying the program area floppy disk, a hard disk, an optical disk, a magneto-optical disk, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, a non-volatile type memorycard, a ROM, and a DVD (DVD-ROM and a DVD-R).

As for the method of supplying the program, a client computer can beconnected to a website on the Internet using a browser of the clientcomputer, and the computer program of the present invention or anautomatically-installable compressed file of the program can bedownloaded to a recording medium such as a hard disk. Further, theprogram of the present invention can be supplied by dividing the programcode constituting the program into a plurality of files and downloadingthe files from different websites. In other words, a WWW (World WideWeb) server that downloads, to multiple users, the program files thatimplement the functions of the present invention by computer is alsocovered by the claims of the present invention.

It is also possible to encrypt and store the program of the presentinvention on a storage medium such as a CD-ROM, distribute the storagemedium to users, allow users who meet certain requirements to downloaddecryption key information from a website via the Internet, and allowthese users to decrypt the encrypted program by using the keyinformation, whereby the program is installed in the user computer.

Besides the cases where the aforementioned functions according to theembodiments are implemented by executing the read program by computer,an operating system or the like running on the computer may perform allor a part of the actual processing so that the functions of theforegoing embodiments can be implemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or a part of the actual processing so thatthe functions of the foregoing embodiments can be implemented by thisprocessing.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2007-279383 filed on Oct. 26, 2007, which is hereby incorporated byreference herein in its entirety.

1-19. (canceled)
 20. A system comprising: a display apparatuscomprising: a first processor; and a first memory having instructionsstored thereon, which, when executed by the first processor, cause thedisplay apparatus to: capture an image of physical space; and transmitthe captured image of physical space; and an image processing apparatuswhich communicates with the display apparatus, the image processingapparatus comprising: a second processor; and a second memory storinginstructions that, when executed by the second processor, cause theimage processing apparatus to: receive the captured image of physicalspace; generate a virtual image based on position and orientationinformation of the display apparatus; generate a synthesized image basedon the captured image of physical space and the generated virtual image;and transmit the generated synthesized image to the display apparatus,wherein the first memory has further instructions stored thereon which,when executed by the first processor, cause the display apparatus to:receive the generated synthesized image; and display the generatedsynthesized image in a first display mode and the captured image ofphysical space in a second display mode, and wherein the first displaymode and the second display mode are switched based on communicationconditions with the image processing apparatus.
 21. The system accordingto claim 20, wherein the image processing apparatus communicateswirelessly with the display apparatus.
 22. The system according to claim20, wherein the second display mode is used to further display thecaptured image of physical space while at least a part of thesynthesized image is not received from the image processing apparatus.23. The system according to claim 20, wherein a message superimposed onthe captured image of physical space is displayed in the second displaymode.
 24. The system according to claim 20, wherein the displayapparatus is used while mounted on a head of a user.
 25. A systemcomprising: a display apparatus comprising: a first processor; and afirst memory having instructions stored thereon which, when executed bythe first processor, cause the display apparatus to: capture an image ofphysical space; and transmit the captured image of physical space; andan image processing apparatus which communicates with the displayapparatus, the image processing apparatus comprising: a secondprocessor; and a second memory having instructions stored thereon,which, when executed by the second processor, cause the image processingapparatus to: receive the captured image of physical space; generate avirtual image based on position and orientation information of thedisplay apparatus; and transmit the generated virtual image to thedisplay apparatus, wherein the first memory has further instructionsstored thereon, which, when executed by the first processor, cause thedisplay apparatus to: receive the generated virtual image; generate asynthesized image based on the captured image of physical space and thegenerated virtual image; and display the generated synthesized image ina first display mode and the captured image of physical space in asecond display mode, and wherein the first display mode and the seconddisplay mode are switched based on communication conditions with theimage processing apparatus.
 26. The system according to claim 25,wherein the image processing apparatus communicates wirelessly with thedisplay apparatus.
 27. The system according to claim 25, wherein thesecond display mode is used to further display the captured image ofphysical space while at least a part of the virtual image is notreceived from the image processing apparatus.
 28. The system accordingto claim 25, wherein a message superimposed on the captured image ofphysical space is displayed in the second display mode.
 29. The systemaccording to claim 25, wherein the display apparatus is used whilemounted on a head of a user.
 30. A display apparatus which communicateswith an image processing apparatus, comprising: a first processor; and afirst memory having instructions stored thereon, which, when executed bythe first processor, cause the display apparatus to: capture an image ofphysical space; and transmit the captured image of physical space,wherein the image processing apparatus comprises: a second processor;and a second memory having instructions stored thereon, which, whenexecuted by the second processor, cause the image processing apparatusto: receive the captured image of physical space; generate a virtualimage based on position and orientation information of the displayapparatus; generate a synthesized image based on the captured image ofphysical space and the generated virtual image; and transmit thegenerated synthesized image to the display apparatus, wherein the firstmemory has further instructions, which, when executed by the firstprocessor, cause the display apparatus to: receive the generatedsynthesized image; and display the generated synthesized image in afirst display mode and the captured image of physical space in a seconddisplay mode, and wherein the first display mode and the second displaymode are switched based on communication conditions with the imageprocessing apparatus.
 31. The display apparatus according to claim 30,wherein the display apparatus communicates wirelessly with the imageprocessing apparatus.
 32. The display apparatus according to claim 30,wherein the second display mode is used to further display the capturedimage of physical space while at least a part of the generatedsynthesized image is not received from the image processing apparatus.33. The display apparatus according to claim 30, wherein a messagesuperimposed on the captured image of physical space is displayed in thesecond display mode.
 34. The display apparatus according to claim 30,wherein the display apparatus is used while mounted on a head of a user.35. A display apparatus which communicates with an image processingapparatus, comprising: a first processor; and a first memory havinginstructions stored thereon, which, when executed by the firstprocessor, cause the display apparatus to: capture an image of physicalspace; and transmit the captured image of physical space, wherein theimage processing apparatus comprises: a second processor; and a secondmemory having instructions stored thereon, which, when executed by thesecond processor, cause the image processing apparatus to: receive thecaptured image of physical space; generate a virtual image based onposition and orientation information of the display apparatus; andtransmit the generated virtual image to the display apparatus, whereinthe first memory has further instructions stored thereon, which, whenexecuted by the first processor, cause the display apparatus to: receivethe generated virtual image; generate a synthesized image based on thecaptured image of physical space and the generated virtual image; anddisplay the generated synthesized image in a first display mode and thecaptured image of physical space in a second display mode, and whereinthe first display mode and the second display mode are switched based oncommunication conditions with the image processing apparatus.
 36. Thedisplay apparatus according to claim 35, wherein the display apparatuscommunicates wirelessly with the image processing apparatus.
 37. Thedisplay apparatus according to claim 35, wherein the second display modeis further used to display the captured image of physical space while atleast a part of the generated virtual image is not received from theimage processing apparatus.
 38. The display apparatus according to claim35, wherein a message superimposed on the captured image of physicalspace is displayed in the second display mode.
 39. The display apparatusaccording to claim 35, wherein the display apparatus is used whilemounted on a head of a user.
 40. A control method of a system comprisinga display apparatus and an image processing apparatus which communicateswith the display apparatus, comprising: at the display apparatus:capturing an image of physical space; and transmitting the capturedimage of physical space; at the image processing apparatus: receivingthe captured image of physical space; generating a virtual image basedon position and orientation information of the display apparatus;generating a synthesized image based on the image of physical space andthe generated virtual image; and transmitting the generated synthesizedimage to the display apparatus; and at the display apparatus: receivingthe generated synthesized image; displaying the generated synthesizedimage in a first display mode and the captured image of physical spacein a second display mode; and switching the first display mode and thesecond display mode based on communication conditions with the imageprocessing apparatus.
 41. A control method of a system comprising adisplay apparatus and an image processing apparatus which communicateswith the display apparatus, comprising: at the display apparatus:capturing an image of physical space; and transmitting the capturedimage of physical space; at the image processing apparatus: receivingthe captured image of physical space; generating a virtual image basedon position and orientation information of the display apparatus; andtransmitting the generated virtual image to the display apparatus; andat the display apparatus: receiving the generated virtual image;generating a synthesized image based on the captured image of physicalspace and the generated virtual image; displaying the generatedsynthesized image in a first display mode and the captured image ofphysical space in a second display mode; and switching the first displaymode and the second display mode based on communication conditions withthe image processing apparatus.